Device with camera-info

ABSTRACT

The present application relates to a device, such as a navigation device, for vehicular and non-vehicular use, e.g. by pedestrians. The device is arranged to receive a feed from a camera. The navigation device is further arranged to display a combination of a camera image from the feed from the camera and virtual signage on the display. The virtual signage relating to roads, buildings, points of interest and the like.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.11/920,730 filed Aug. 19, 2008; the entire contents of which isincorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates to a navigation device, the navigationdevice being arranged to display navigation directions on a display.

Also, the present invention relates to a vehicle comprising such anavigation device, and a method for providing navigation directions.Moreover, the present invention relates to a computer program and a datacarrier.

STATE OF THE ART

Prior art navigation devices based on GPS (Global Positioning System)are well known and are widely employed as in-car navigation systems.Such a GPS based navigation device relates to a computing device whichin a functional connection to an external (or internal) GPS receiver iscapable of determining its global position. Moreover, the computingdevice is capable of determining a route between start and destinationaddresses, which can be input by a user of the computing device.Typically, the computing device is enabled by software for computing a“best” or “optimum” route between the start and destination addresslocations from a map database. A “best” or “optimum” route is determinedon the basis of predetermined criteria and need not necessarily be thefastest or shortest route.

The navigation device may typically be mounted on the dashboard of avehicle, but may also be formed as part of an on-board computer of thevehicle or car radio. The navigation device may also be (part of) ahand-held system, such as a PDA.

By using positional information derived from the GPS receiver, thecomputing device can determine at regular intervals its position and candisplay the current position of the vehicle to the user. The navigationdevice may also comprise memory devices for storing map data and adisplay for displaying a selected portion of the map data.

Also, it can provide instructions how to navigate the determined routeby appropriate navigation directions displayed on the display and/orgenerated as audible signals from a speaker (e.g. ‘turn left in 100 m’).Graphics depicting the actions to be accomplished (e.g. a left arrowindicating a left turn ahead) can be displayed in a status bar and alsobe superimposed upon the applicable junctions/turnings etc. in the mapitself.

It is known to enable in-car navigation systems to allow the driver,whilst driving in a car along a route calculated by the navigationsystem, to initiate a route re-calculation. This is useful where thevehicle is faced with construction work or heavy congestion.

It is also known to enable a user to choose the kind of routecalculation algorithm deployed by the navigation device, selecting forexample from a ‘Normal’ mode and a ‘Fast’ mode (which calculates theroute in the shortest time, but does not explore as many alternativeroutes as the Normal mode).

It is also known to allow a route to be calculated with user definedcriteria; for example, the user may prefer a scenic route to becalculated by the device. The device software would then calculatevarious routes and weigh more favourably those that include along theirroute the highest number of points of interest (known as POIs) tagged asbeing for example of scenic beauty.

In the prior art, navigation devices display maps that are, like mostmaps, a highly stylised or schematic representation of the real world.Many people find it difficult to translate this quite abstract versionof the real world into something that can readily be recognised andunderstood. Navigation devices are known that display a (semi) threedimensional projection of the map, as would be seen from above and/orbehind the vehicle. This is done to make it easier for the user tointerpret the displayed map data, as it corresponds to the user's visualperception of the world. However, such a (semi) perspective view is astylised or schematic representation that still is relatively difficultto be interpreted by users.

Yet the need to enable people to easily and rapidly follow directionsthat are shown on the display is especially acute in a personalnavigation system, such as may be used as an in-car navigation system.It will be understood that a driver of a vehicle should spend as littletime as possible watching and interpreting the displayed map data, ashis/her main attention should be focussed on the road and the traffic.

SHORT DESCRIPTION OF THE INVENTION

Therefore, it is an object of the invention to provide a navigationdevice that overcomes at least one of the problems mentioned above anddisplays instructions for the user that allow easy interpretation.

In order to obtain this object, the invention provides a navigationdevice according to the preamble, characterised in that, the navigationdevice is further arranged to receive a feed from a camera, and thenavigation device being arranged to display a combination of a cameraimage from the feed from the camera and the navigation directions on thedisplay.

By superimposing or combining navigation directions over a camera image,a user-friendly view is presented to the driver that allows easy andrapid interpretation. There is no need for the user to translate anabstract representation of the real-world, because the camera image is aone-to-one representation of the real-life view as seen by the user. Thecombination of the feed from the camera and the navigation directionscould be all kinds of combinations, such as superimposing one over theother, showing simultaneously on different parts of the display. Thecombination may however also be a combination in time, i.e. alternatelyshowing the camera feed and the navigation directions. This may changeafter a predetermined time interval (e.g. 5 seconds) or may change as aresult of an input by the user.

According to a further embodiment, the invention relates to a navigationdevice, wherein the camera is formed integrally with the navigationdevice. Such a navigation device doesn't require an external camerafeed. The navigation device can for instance simply be mounted on adashboard of a vehicle, in such a way that the camera provides an imagethrough the front screen.

According to a further embodiment, the invention relates to a navigationdevice wherein the navigation directions are one or more of positionarrow, route, arrow, points of interest, roads, buildings, map data suchas vector data, stored in at least a memory unit, such as a hard disk, aRead Only Memory, Electrically Erasable Programmable Read Only Memoryand a Random Access Memory. All kind of navigation directions can bedisplayed. It is noted that these navigations directions may alsoprovide information that is not per se needed for navigation (finding aroute), but may also provide the user with additional information.

According to a further embodiment, the invention relates to a navigationdevice further being arranged to superimpose the navigation directionsover the camera image such that the position of the navigationdirections are in a predefined spatial relationship with respect tocorresponding parts of the camera image. This provides the user with animage that can very easy be interpreted, as all the navigationdirections may be displayed such that they match with the actualposition of the corresponding item in the camera image. For instance, anarrow indicating a right turn may be superimposed over the camera imagesuch that it matches with the turn as visible in the camera image.

According to a further embodiment, the invention relates to a navigationdevice, wherein the navigation device comprises a processing unit, apositioning device and orientation sensors, the positioning device andthe orientation sensors being arranged to communicate with theprocessing unit, the processing unit being arranged to use readings fromthe positioning device and the orientation sensors to compute a positionand an orientation of the camera and/or the navigation device, based onwhich the position of the navigation directions on the display arecomputed by the processing unit. Knowing the exact position andorientation of the camera and/or the navigation device allows more exactsuperimposing of the navigation directions over the camera feed.

According to a further embodiment, the invention relates to a navigationdevice, wherein the positioning device determines a geographicallocation, using positioning sensing technology, such as GPS, EuropeanGalileo system or any other global navigation satellite system, orpositioning sensing technology based on ground-based beacons.

According to a further embodiment, the invention relates to a navigationdevice, wherein the processing unit computes the orientation of thecamera with respect to a first rotational axis that in use issubstantially vertical, by comparing the positions of the camera and/orthe navigation device determined by the positioning device at subsequentpoints in time. By comparing the positions of the camera and/ornavigation device at subsequent points in time, the direction of travelof the camera and/or navigation device can be computed. From this, theorientation and the change of orientation of the camera can be computed.

According to a further embodiment, the invention relates to a navigationdevice, wherein the navigation device comprises a compass providingcompass readings to the processing unit, the processing unit beingarranged to compute the orientation of the camera with respect to afirst rotational axis that in use is substantially vertical, based onthe compass readings. A compass provides an easy an advantageous way ofdetermining the orientation of the camera.

According to a further embodiment, the invention relates to a navigationdevice, wherein the orientation sensors comprise tilt sensors todetermine the orientation of the camera with respect to second and thirdrotational axes, the second and third rotational axes in use beingsubstantially horizontal. In order to combine or superimpose thenavigation directions in a more accurate way with respect to the cameraimage, the rotational orientation of the camera is measured with respectto a second and/or third direction.

According to a further embodiment, the invention relates to a navigationdevice, wherein the processing unit uses pattern recognition techniquesto superimpose the navigation directions over the camera image such thatthe position of the navigation directions are in a predefined spatialrelationship with respect to corresponding parts of the camera image. Byusing pattern recognition techniques, the navigation directions can becombined and/or superimposed over the camera feed without knowing theexact orientation of the camera. Determining the position of thenavigation directions over the displayed camera image may be done bysolely using pattern recognition techniques, but the pattern recognitiontechniques may also be used in combination with a determined orientationof the camera, to further increase the accuracy.

According to a further embodiment, the invention relates to a navigationdevice, wherein the navigation device uses map data as input for thepattern recognition techniques. Using map data may simplify the patternrecognition techniques, as it is easier to recognise for instance aroad, when it is approximately known from the map data where the roadis. This makes the pattern recognition more accurate and/or may savecomputation time.

According to a further embodiment, the invention relates to a navigationdevice, wherein the navigation device is arranged to receive calibrationcorrections, to store these calibration corrections, and to apply thecalibration corrections when combining the navigation directions and thecamera image. This is in particular advantageous when the navigationdirections are combined in such a way, that the navigation directionsare superimposed over the camera image to have a predefined spatialrelationship with respect to the camera image. The calibrationscorrections may be used to cancel offset errors.

According to a further embodiment, the invention relates to a navigationdevice, wherein the navigation device is arranged to receive or read incamera settings and use the camera settings to compute the position ofthe navigation directions on the display. Different camera settings mayresult in different camera feeds. Providing the navigation device withthese camera settings further increases the accuracy of the combinationof the navigation directions with the camera image.

According to a further embodiment, the invention relates to a navigationdevice, wherein the navigation device is further arranged to receivefeeds from more than one camera, and the navigation device beingarranged to select one of the feeds to be displayed on the display. Themore than one camera feeds, providing different perspectives, may forinstance be used by pattern recognition techniques to increase thequality of pattern recognition using mathematics. The more than onecamera may also be used to provide the user with the option of choosingbetween different camera angles.

According to a further embodiment, the invention relates to a navigationdevice, wherein the camera is sensitive to electromagnetic radiationoutside the range of the electromagnetic spectrum that is visible by thehuman eye.

According to a further embodiment, the invention relates to a navigationdevice, wherein the camera is an infrared camera. Such a camera enablesuse of the navigation device at night.

According to a further embodiment, the invention relates to a navigationdevice, wherein the camera is arranged to zoom in and/or to zoom out.This allows the user to adjust the camera view according to his or herpreferences.

According to a further embodiment, the invention relates to a navigationdevice, wherein the camera is arranged to zoom in or out depending on,for instance, the speed of the navigation device/vehicle. This providesa camera feed that is automatically adjusted to the speed of thenavigation device. So, in case the speed of the navigation device isrelatively high, the camera may zoom in to give the user a better viewfurther ahead.

According to a further aspect, the invention relates to a dashboard,comprising a navigation device according to the above.

According to a further aspect, the invention relates to a vehicle,comprising a navigation device according to the above.

According to a further embodiment, the invention relates to a vehicle,wherein the vehicle comprises a vehicle tilt sensor to determine thetilt of the vehicle, providing vehicle tilt readings to the navigationdevice. This is an advantageous way of measuring the tilt of thevehicle.

According to a further aspect, the invention relates to a method forproviding navigation directions, the method comprising:

-   -   displaying navigation directions on a display, characterised in        that, the method further comprises:    -   receiving a feed from a camera, and    -   displaying a combination of a camera image from the feed from        the camera and the navigation directions over the camera image        on the display.

According to a further aspect, the invention relates to a computerprogram, when loaded on a computer arrangement, arranged to perform theabove method.

According to a further aspect, the invention relates to a data carrier,comprising a computer program as described above.

SHORT DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will now be described, by way of exampleonly, with reference to the accompanying schematic drawings in whichcorresponding reference symbols indicate corresponding parts, and inwhich:

FIG. 1 schematically depicts a schematic block diagram of a navigationdevice,

FIG. 2 schematically depicts a schematic view of a navigation device,

FIG. 3 schematically depicts a schematic block diagram of a navigationdevice according an embodiment of the invention,

FIG. 4 schematically depicts a vehicle comprising a navigation deviceaccording to an embodiment of the invention,

FIG. 5 schematically depicts a navigation device according to anembodiment of the invention,

FIG. 6 schematically depicts a navigation device according to anembodiment of the invention,

FIG. 7 schematically depicts a camera according to an embodiment of theinvention,

FIGS. 8 a and 8 b schematically depict different movement of the cameraimage on the display as a result of different tilts of the camera,

FIG. 9 schematically depicts a flow diagram of the functionality of thenavigation device 10 according to an embodiment of the invention,

FIG. 10 schematically depicts a navigation device according to anembodiment of the invention,

FIG. 11 depicts a navigation device according to an embodiment of theinvention, and

FIG. 12 depicts a navigation device according to a further embodiment ofthe invention.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 shows a schematic block diagram of an embodiment of a navigationdevice 10, comprising a processor unit 11 for performing arithmeticaloperations. The processor unit 11 is arranged to communicate with memoryunits that store instructions and data, such as a hard disk 12, a ReadOnly Memory (ROM) 13, Electrically Erasable Programmable Read OnlyMemory (EEPROM) 14 and a Random Access Memory (RAM) 15. The memory unitsmay comprise map data 22. This map data may be two dimensional map data(latitude and longitude), but may also comprise a third dimensions(height). The map data may further comprise additional information suchas information about petrol/gas stations, points of interest. The mapdata may also comprise information about the shape of buildings andobjects along the road.

The processor unit 11 may also be arranged to communicate with one ormore input devices, such as a keyboard 16 and a mouse 17. The keyboard16 may for instance be a virtual keyboard, provided on a display 18,being a touch screen. The processor unit 11 may further be arranged tocommunicate with one or more output devices, such as a display 18, aspeaker 29 and one or more reading units 19 to read for instance floppydisks 20 or CD ROM's 21. The display 18 could be a conventional computerdisplay (e.g. LCD) or could be a projection type display, such as thehead up type display used to project instrumentation data onto a carwindscreen or windshield. The display 18 may also be a display arrangedto function as a touch screen, which allows the user to inputinstructions and/or information by touching the display 18 with hisfinger.

The processor unit 11 may further be arranged to communicate with othercomputing devices or communication devices using an input/output device25. The input/output device 25 is shown to be arranged to equipcommunication via a network 27.

The speaker 29 may be formed as part of the navigation device 10. Incase the navigation device 10 is used as an in-car navigation device,the navigation device 10 may use speakers of the car radio, the boardcomputer and the like.

The processor unit 11 may further be arranged to communicate with apositioning device 23, such as a GPS receiver, that provides informationabout the position of the navigation device 10. According to thisembodiment, the positioning device 23 is a GPS based positioning device23. However, it will be understood that the navigation device 10 mayimplement any kind of positioning sensing technology and is not limitedto GPS. It can hence be implemented using other kinds of GNSS (globalnavigation satellite system) such as the European Galileo system.Equally, it is not limited to satellite based location/velocity systemsbut can equally be deployed using ground-based beacons or any other kindof system that enables the device to determine its geographicallocation.

However, it should be understood that there may be provided more and/orother memory units, input devices and read devices known to personsskilled in the art. Moreover, one or more of them may be physicallylocated remote from the processor unit 11, if required. The processorunit 11 is shown as one box, however, it may comprise several processingunits functioning in parallel or controlled by one main processor thatmay be located remote from one another, as is known to persons skilledin the art.

The navigation device 10 is shown as a computer system, but can be anysignal processing system with analog and/or digital and/or softwaretechnology arranged to perform the functions discussed here. It will beunderstood that although the navigation device 10 is shown in FIG. 1 asa plurality of components, the navigation device 10 may be formed as asingle device.

The navigation device 10 may use navigation software, such as navigationsoftware from TomTom B.V. called Navigator. Navigator software may runon a touch screen (i.e. stylus controlled) Pocket PC powered PDA device,such as the Compaq iPaq, as well as devices that have an integral GPSreceiver 23. The combined PDA and GPS receiver system is designed to beused as an in-vehicle navigation system. The invention may also beimplemented in any other arrangement of navigation device 10, such asone with an integral GPS receiver/computer/display, or a device designedfor non-vehicle use (e.g. for walkers) or vehicles other than cars (e.g.aircraft).

FIG. 2 depicts a navigation device 10 as described above.

Navigator software, when running on the navigation device 10, causes anavigation device 10 to display a normal navigation mode screen at thedisplay 18, as shown in FIG. 2. This view may provide drivinginstructions using a combination of text, symbols, voice guidance and amoving map. Key user interface elements are the following: a 3-D mapoccupies most of the screen. It is noted that the map may also be shownas a 2-D map.

The map shows the position of the navigation device 10 and its immediatesurroundings, rotated in such a way that the direction in which thenavigation device 10 is moving is always “up”. Running across the bottomquarter of the screen may be a status bar 2. The current location of thenavigation device 10 (as the navigation device 10 itself determinesusing conventional GPS location finding) and its orientation (asinferred from its direction of travel) is depicted by a position arrow3. A route 4 calculated by the device (using route calculationalgorithms stored in memory devices 11, 12, 13, 14, 15 as applied to mapdata stored in a map database in memory devices 11, 12, 13, 14, 15) isshown as darkened path. On the route 4, all major actions (e.g. turningcorners, crossroads, roundabouts etc.) are schematically depicted byarrows 5 overlaying the route 4. The status bar 2 also includes at itsleft hand side a schematic icon depicting the next action 6 (here, aright turn). The status bar 2 also shows the distance to the next action(i.e. the right turn—here the distance is 50 meters) as extracted from adatabase of the entire route calculated by the device (i.e. a list ofall roads and related actions defining the route to be taken). Statusbar 2 also shows the name of the current road 8, the estimated timebefore arrival 9 (here 2 minutes and 40 seconds), the actual estimatedarrival time 25 (11.36 am) and the distance to the destination 26 (1.4Km). The status bar 2 may further show additional information, such asGPS signal strength in a mobile-phone style signal strength indicator.

As already mentioned above, the navigation device may comprise inputdevices, such as a touch screen, that allows the users to call up anavigation menu (not shown). From this menu, other navigation functionscan be initiated or controlled. Allowing navigation functions to beselected from a menu screen that is itself very readily called up (e.g.one step away from the map display to the menu screen) greatlysimplifies the user interaction and makes it faster and easier. Thenavigation menu includes the option for the user to input a destination.

The actual physical structure of the navigation device 10 itself may befundamentally no different from any conventional handheld computer,other than the integral GPS receiver 23 or a GPS data feed from anexternal GPS receiver. Hence, memory devices 12, 13, 14, 15 store theroute calculation algorithms, map database and user interface software;a processor unit 12 interprets and processes user input (e.g. using atouch screen to input the start and destination addresses and all othercontrol inputs) and deploys the route calculation algorithms tocalculate the optimal route. ‘Optimal’ may refer to criteria such asshortest time or shortest distance, or some other user-related factors.

More specifically, the user inputs his start position and requireddestination into the navigation software running on the navigationdevice 10, using the input devices provided, such as a touch screen 18,keyboard 16 etc. The user then selects the manner in which a travelroute is calculated: various modes are offered, such as a ‘fast’ modethat calculates the route very rapidly, but the route might not be theshortest; a ‘full’ mode that looks at all possible routes and locatesthe shortest, but takes longer to calculate etc. Other options arepossible, with a user defining a route that is scenic—e.g. passes themost POI (points of interest) marked as views of outstanding beauty, orpasses the most POIs of possible interest to children or uses the fewestjunctions etc.

Roads themselves are described in the map database that is part ofnavigation software (or is otherwise accessed by it) running on thenavigation device 10 as lines—i.e. vectors (e.g. start point, end point,direction for a road, with an entire road being made up of many hundredsof such sections, each uniquely defined by start point/end pointdirection parameters). A map is then a set of such road vectors, pluspoints of interest (POIs), plus road names, plus other geographicfeatures like park boundaries, river boundaries etc, all of which aredefined in terms of vectors. All map features (e.g. road vectors, POIsetc.) are defined in a co-ordinate system that corresponds or relates tothe GPS co-ordinate system, enabling a device's position as determinedthrough a GPS system to be located onto the relevant road shown in amap.

Route calculation uses complex algorithms that are part of thenavigation software. The algorithms are applied to score large numbersof potential different routes. The navigation software then evaluatesthem against the user defined criteria (or device defaults), such as afull mode scan, with scenic route, past museums, and no speed camera.The route which best meets the defined criteria is then calculated bythe processor unit 11 and then stored in a database in the memorydevices 12, 13, 14, 15 as a sequence of vectors, road names and actionsto be done at vector end-points (e.g. corresponding to pre-determineddistances along each road of the route, such as after 100 meters, turnleft into street x).

FIG. 3 depicts a schematic block diagram of a navigation device 10according to the invention, in which corresponding reference symbolsrefer to corresponding parts as in FIGS. 1 and 2.

According to the invention a camera 24 is provided that is arranged toprovide a real time feed to the processor unit 11. The camera 24 is, inuse, positioned such that it registers the road ahead of the user. Whenpositioned in a car, the camera 24 is positioned such that it registersthe road ahead of the vehicle. The camera 24 may be integral with thenavigation device 10, or may be physically separate from it. Ifseparate, the camera 24 may be connected to the processor unit 11 viacabling or via a wireless connection. The camera 24 may be positioned onthe roof of the vehicle or at the front of the vehicle, for instanceclose to the headlights.

The navigation device 10 may also be provided with more than one camera24, to allow the user to switch between different camera angles. Also arear view camera may be provided. The camera may be any type of camera,such as a digital camera or an analogue camera. The image as registeredby the camera 24 is displayed at the display 18.

The camera 24 may also be a camera that is sensitive to electro-magneticradiation outside the electro-magnetic spectrum that is visible by thehuman eye. The camera may be an infrared camera that enables use atnight.

FIG. 4 shows an example of a navigation device 10, positioned on thedashboard of a car 1. The navigation device 10 comprises a camera 24that is directed at the road ahead of the car 1. FIG. 4 further showsthat the display 18 faces the user.

According to the invention, the navigation device 10 is arranged todisplay the real time feed from the camera on the display 18 and tocombine or superimpose one or more navigation directions. The navigationdirections may be one or more of the following: position arrow 3, theroute 4, arrow 5, points of interest, roads, buildings and all furthernavigation directions stored in the navigation device 10. This may alsoinclude the map data itself, e.g. the vector data describing the roads.A more detailed description of how this is achieved follows below.

The images provided by the camera 24 will not be steady, due to thebumpiness of the road, vibrations of the vehicle caused by the engineetc. Therefore, the navigation device 10 may be provided with softwarethat cancels these unwanted vibrations to provide a steady image.Software that cancels unwanted vibrations of the images provided by thecamera 24 is widely used in video cameras, where it is used under thename steady cam. This is known to a skilled person.

The feed from the camera 24 may further be processed to increase thequality of the images. This processing may comprise adjusting thebrightness, contrast, but may be any suitable filter. Filters may beused to increase the quality of the images in rainy conditions.

The feed from the camera 24 can be displayed on the display inreal-time, but may also be displayed as a still that is updated atcertain points in time, for instance every 0.5 seconds. The appropriatetime intervals between successive updates may be determined independence of the speed of the navigation device 10 vehicle, change ofdirection of travel (taking bends).

Also, the navigation device may be arranged to perform zoom in or outdepending on, for instance, the speed of the navigation device/vehicle.This zoom operation may be performed by sending a control signal to thecamera 24 giving it instructions to perform a zoom operation. The zoomoperation may however also be performed by displaying a part of thereceived camera feed in an enlarged way at the display 18.

Embodiment 1

FIG. 5 depicts a first example of the invention. FIG. 5 shows a still ofthe image registered by the camera 24 as displayed by the navigationdevice 10. As can be seen, an arrow 5 indicating a right turn issuperimposed by the processor unit 11. According to this embodiment, auser-friendly image is displayed to the user, allowing easyinterpretation. This embodiment has the advantage that no complexmathematics and data processing is needed.

Instead of the navigation direction depicted in FIG. 5, also othernavigation directions as mentioned above may be displayed, includingperspective shaped navigation directions, such as perspective shapedarrows.

Embodiment 2

FIG. 6 shows another still of the image registered by the camera 24.According to this example, the navigation device 10 superimposes theroute 4 and the arrow 5. The route 4 and the arrow 5 are superimposed insuch a way that their position on the display 18 corresponds with theimage as provided by the camera 24. FIG. 6 clearly shows that the route4 is displayed such that it corresponds with the road as shown on thedisplay 18. Also, the arrow 5 is displayed in such a way that itaccurately indicates a right turn in the image as provided by the camera24.

It will be understood that the embodiment shown in FIG. 5 can easily beobtained by superimposing or combining the image as provided by thecamera 24 and a navigation direction, as for instance the arrow 5.However, in order to create the image as provided in FIG. 6, morecomplicated data processing is required in order to match the image asprovided by the camera 24 with the navigation directions. This will beexplained in more detail below.

In order to superimpose the navigation directions such that it has apredefined spatial relationship with respect to corresponding parts ofthe camera image, the exact camera position, direction and camerasettings need to be known. If all this information is known, theprocessing unit 11 computes the position of for instance the road on thedisplay 18 and superimposes the route 4.

First, the position of the camera 24 needs to be determined. This maysimply be done by using the GPS information as determined by theprocessing unit 11 and/or the positioning device 23. The positioninformation of the navigation device 10, and thus the camera 24, isalready available in the navigation device 10 according to prior artuse.

Second, the orientation of the camera 24 needs to be determined. This isdone using orientation sensors, arranged to communicate with theprocessing unit 11. The orientation sensors may be the positioningdevice 23 and tilt sensors 27, 28. The tilt sensors 27, 28 may begyroscopes.

FIG. 7 depicts a camera 24 according to an embodiment of the invention.A first rotational direction needs to be determined with respect to anaxis C, as depicted in FIG. 7. Also, this may simply be done using theGPS information as determined by the processing unit 11 and/or thepositioning device 23. By comparing the position of the navigationdevice 10 at successive points in time, the direction of movement of thenavigation device 10 can be determined. This information is also alreadyavailable in the navigation device 10 according to prior art use. It isassumed that the camera 24 faces in the direction of travel of thenavigation device 10. However, this is not necessarily the case, as willbe further explained below.

The first rotational direction C of the camera 24 may also be determinedby using a (electronic) compass comprised by the navigation device orcamera 24. The compass may be an electronic compass or an analoguecompass. The compass provides compass readings that are communicated tothe processing unit 11. Based on the compass readings the processingunit 11 determines the first rotational direction of the camera 24.

In order to further determine the orientation of the camera 24, thecamera 24 may be provided with tilt sensors 27, 28 as depicted by FIG.7. The tilt sensors 27, 28 are arranged to measure the tilt of thecamera 24. The first tilt sensor 27 is arranged to measure the tilt in asecond rotational direction as indicated by the curved arrow A in FIG.7, i.e. a rotation about an axis being substantially perpendicular tothe drawings surface. The tilt in the second rotational directiondetermines the height of the horizon in the camera image as displayed onthe display 18. The effect of such a rotation on the camera image asdisplayed is schematically depicted in FIG. 8 a.

The second tilt sensor 28 is arranged to measure the tilt as a result ofa rotation about a third rotational axis, being a central axis, of thecamera 24 depicted in FIG. 7 by the dotted line B. The effect of such arotation on the camera image as displayed is schematically depicted inFIG. 8 b.

In use, the first rotational axis is substantially vertical and thesecond and third rotational axes are substantially perpendicular withrespect to the first rotational axis and with respect to each other.

The tilt values as determined by the tilt sensors 27, 28 arecommunicated to the processor unit 11. Tilt sensors 27 and 28 may alsobe formed as a single integral tilt sensor.

Also, the camera settings, in particular the zoom factor of the lens ofthe camera 24, camera angle, focal length etc., may be communicated tothe processor unit 11.

Based on the information available to the processor unit 11 fordescribing the position, direction and settings of the camera 24, theprocessor unit 11 determines the position where the road, crossings,forks, points of interest etc. corresponding to the map data stored inthe memory devices 11, 12, 13, 14, 15 are to be displayed at the display18.

Based on this information, the processor unit 11 may superimposenavigation directions, such as the route 4, the arrow 5, points ofinterest POI etc. over the camera image as displayed by the processorunit 11, such that they coincide with the camera view. It may be usefulto superimpose the navigation directions so that they appear to floatabove the road surface or have some other pre-defined spatialrelationship to it.

Since the navigation device 10 computes how far away any junction orturning (or other directional change) is, it can work out approximatelyhow a navigation direction displayed on the display 18 should be shapedand where it should be positioned in order to correspond to the actuallocation of the change in direction as shown on the feed from the camera24.

However, errors may occur because of several reasons. In the firstplace, the navigation device 10 can be mounted on the dashboard of avehicle in many ways. For instance, when determining the firstrotational direction of the camera 24 with respect to the axis C bycomparing positions of the navigation device 24 at successive points intime, it is assumed, that the camera 24 is directed straight ahead.However, in case the camera 24 is not perfectly aligned with thevehicle, a mismatch of the superimposed navigation directions may occur.

As discussed above, in case the camera 24 is provided with a built-incompass, the first rotational orientation of the camera with respect toaxis C can be computed by comparing the compass readings with thedetermined direction of travel of the navigation device 10. However,still an error may be present resulting in a mismatch between thesuperimposed navigation directions and the camera feed.

Also, the tilt sensors 27, 28 may be only capable of measuring relativetilt, and not absolute tilt. This means that the navigation device 10needs to be calibrated in order to allow accurate positioning of thenavigation directions over the camera image.

In order to compensate for these errors, the navigation device 10 may beprovided with a menu option that allows the user to adjust the relativeposition of the displayed image with respect to the displayed cameraimage. This adjustment may be carried out by the navigation device 10 bychanging the position where the navigation directions are displayed,and/or by changing the position where the camera image is displayed,and/or by changing the orientation of the camera 24. For the lastoption, the camera 24 may be provided with an actuation device to changeits orientation. The camera 24 may be actuated independent of thenavigation device 10. In case the camera 24 is integrally formed withthe navigation device 10, the actuation device may change theorientation of the navigation device 10, or of the camera 24 only withrespect to the navigation device 10.

The user may simply use arrow keys to calibrate the position of thenavigation directions to make them matching with the camera image. Forinstance, if the camera 24 is positioned in such a way, that it istilted to the left about the axis C as depicted in FIG. 7, thenavigation directions are right from the corresponding parts in thecamera image. The user can simply correct for this error by using a leftkey arrow to drag the navigation directions to the left. The navigationdevice 10 may further be arranged to provide the user with options toadjust the displayed rotational orientation of the superimposednavigation directions with respect to the displayed camera image.

The navigation device 10 may also be arranged to provide the user withoptions to correct for perspective mismatching, for instance caused bydifferent heights of the camera 24. A camera 24 positioned on top of acar provides a different view of the road (different perspective shape)than a camera 24 positioned on the dashboard or between the headlightsof a vehicle. In order to make the navigation directions, such as 3Ddirections (e.g. a 3D arrow) or the vector representation of the road,to fit the camera view, a perspective deformation of the navigationdirections need to be applied. This perspective deformation depends fromthe height of the camera 24, the camera settings and the secondrotational direction of the camera 24 in the direction of arrow A asdepicted in FIG. 7.

The processor unit 11 stores these inputted calibration corrections andapplies similar calibration corrections to all further displayed images.All further changes in the measured position, direction and orientationof the camera 24 can be processed by the processor unit 11 tocontinuously ensure accurate superimposing of the navigation directions.This allows accurate compensation of camera movements caused change ofdirection of the vehicle, or caused by speed ramps, sharp corners,accelerations, braking etc. and other causes influencing the orientationof the camera 24.

FIG. 9 depicts a flow diagram depicting the functionality of thenavigation device 10 according to the second embodiment of theinvention. The steps shown in the flow diagram may be performed by theprocessing unit 11. It is noted that all steps relating to the inputtingof a destination address, selecting a route etc. are omitted in thisfigure as these steps are already known in the prior art.

In a first step 101, the navigation device 10 is switched on and theuser selects the camera modus. This is depicted in FIG. 9 with “start”.

In a second step 102, the processing unit 11 determines the position ofthe navigation device 10. This is done by using input from thepositioning device 23, such as a GPS device, as discussed above.

In a next step 103, the processing unit 11 determines the direction oftravel of the navigation device 10. Again, input from the positioningdevice 23 is used for this.

Next, in step 104, the orientation of the camera 24 and the camerasettings are determined by the processing unit 11. Again, input is usedfrom the positioning device 23. Input is also used from the tilt sensors27, 28 to determine the orientation of the camera 24.

According to step 105, the camera image is displayed on the display 18by the processing unit 11. In step 106, the processing unit 11superimposes a selected number of navigation directions (such asposition arrow 3, route 4, arrow 5, points of interest, roads, map dataetc.). In order to do this, all collected information is used to computethe position and shape of the displayed navigation directions. Ifneeded, the user may calibrate this computation by adjusting theposition and/or shape of the superimposed navigation directions. Thisoptional step is depicted by step 107.

Steps 102-107 may be repeated as often as needed or desired during use.

Other kinds of virtual signage in addition to direction arrows 5 mayalso be stored in memory devices 12, 13, 14, 15. For example, iconsrelating to road names, traffic signs, speed limits, speed cameras, orpoints of interest stored in memory devices 12, 13, 14, 15 may bestored. All of these can also be superimposed over the feed from thecamera 24, with a spatial location in the displayed camera image thatcorresponds to the real world feature that the virtual signage relatesto. Hence, the processing unit 11 could take the 2D map data from thenavigation software that included the location data for these real worldfeatures, and apply a geometrical transformation that causes them to becorrectly located when superimposed in the video feed.

In case e.g. a vehicle carrying a navigation device 10 drives up or downa hill, the tilt sensors 27, 28 detect a tilt in the direction of arrowA as depicted in FIG. 7. However, in order to correctly superimpose thenavigation directions over the camera image such that the navigationdirections coincide with the camera image, this tilt should not becorrected for. This can be arranged by providing the navigation devicewith map data comprising height information. Based on the map heightdata, the navigation device 10 computes the tilt of the camera 24 thatcorresponds with the orientation of the road the vehicle is travelingon. This predicted tilt is compared with the tilt as detected by thetilt sensors 27, 28. The difference between the predicted tilt and thedetected tilt is used to adjust the position of the superimposednavigation directions.

In case the map data doesn't comprise height information, the vehiclemay be provided with a vehicle tilt sensor 30. The vehicle tilt sensor30 is arranged to provide vehicle tilt readings to the processing unit11. The readings of the vehicle tilt sensor 30 are then compared withthe readings of the tilt sensors 27, 28 and the difference, caused byunwanted vibrations etc., is used to adjust the position of thesuperimposed navigation directions.

It will be understood that all kinds of variations to the aboveexplained and shown example can be thought of.

FIG. 10 depicts an example in which the map data also comprises datadescribing objects along the road, such as buildings 31. According tothis example, the navigation directions 3, 4, 5 that are superimposedover a building 31 can be shown by dashed or blinking lines. This allowsa user to visualize map data, route 4 and arrows 5 that would otherwisebe blocked from sight by a building.

Third Embodiment

According to a third embodiment, the navigation directions aresuperimposed over the camera image by using pattern recognitiontechniques.

In recent years, considerable progress has been made in the field ofreal time analysis of image frames (e.g. a video feed such as providedby camera 24) to identify actual objects in the video feed. Theliterature is quite extensive in this area: reference may for example bemade to U.S. Pat. No. 5,627,915 (Princeton Video Image Inc.) in whichvideo from a scene such as a sports stadium is analysed by patternrecognition software; an operator manually indicates high contrast areasin the stadium (e.g. lines marked on the playing surface; edges of theplaying surface; billboards) and the software builds up a geometricalmodel of the entire stadium using these high contrast landmarks. Then,the software is able to analyse a real time video feed looking for theselandmarks; it is then able to take a stored computer generated image(e.g. an advertisement for a billboard), apply a geometrical transformto the stored image so that, when inserted into the video feed at alocation defined with reference to the geometrical model using imagesynthesis techniques, it appears to be an entirely natural part of thescene to a viewer of the video.

Reference may also be made to US 2001/0043717 to Facet Technology; thisdiscloses a system that can analyse video taken from a moving vehicle torecognise road signs.

Overall, the pattern recognition arts applied to the analysis of realtime video in order to recognise real world features is a large and wellestablished field.

In one implementation, the navigation device 10 deploys patternrecognition software to recognise real world features in the video feedfrom the camera 24 and displays the navigation directions (such as arrow5) on the display 18 in a pre-defined spatial relationship to the realworld features recognised in the video feed. For example, the video feedmight show the current road that the navigation device 10 is travellingalong and the navigation directions are then 3D directions (e.g. a 3Darrow) that are superimposed over that road. Road turnings and otherfeatures can be graphically or iconically represented and be positionedto overlie the real world features that they relate to.

The processing unit 11 may be programmed so that it can recognisefeatures with a high visual contrast and that are associated with agiven road. The features could also be vehicles moving in a consistentdirection or road markings (e.g. edge markings, centre line markingsetc.).

It is noted that the navigation device 10 is programmed so that it canrecognise features with a high visual contrast and that are associatedwith a road. For example, the features could be vehicles moving in aconsistent direction, or road markings.

The navigation device 10 could for example be programmed with ageometrical model of the road ahead: the model can be as simple as twolines. The model may just be the vector data stored to form the mapdata, as described above.

Then, in use, the pattern recognition software looks for visual featuresin the real time video stream provided by the camera 24 that correspondto the stored geometrical model (e.g. the two lines). Once it haslocated these features, it has in effect recognised the road ahead. Thiswill typically require rapid translations and transformation to beapplied to the features recognised in the video feed (e.g. the twolines) to get a match to the stored model; the translations are x-ytranslations in order to approximately align the recognised featureswith the stored model. The transformations include foreshortening tocorrespond to different camera heights and relative orientation betweenthe two lines to correspond to different camera viewing angles and therelative angle between camera and road. Equally, the transformations canbe applied to align and shape the stored model to the recognisedfeatures.

It will be understood by a skilled person it is advantageous for thepattern recognition algorithm to have the map data as an input.Recognizing a pattern can be done in an easier and faster way when thealgorithm has knowledge beforehand about the patterns to recognize. Thisknowledge can easily be obtained from the available map data.

Once the transformation is known, it is a relatively simple matter ofshaping a pre-stored arrow icon so that it's perspective, shape ororientation corresponds to that of the road in any given video frame(various kinds of geometrical transforms may be suitable for this) andthen superimposing the directional arrow over the road shown in thedisplay using conventional image synthesis. It may be useful tosuperimpose the arrow so that it appears to float above the road surfaceor have some other pre-defined spatial relationship to it.

Since the navigation device 10 computes how far away any junction orturning (or other directional change) is, it can work out approximatelyhow a navigation direction displayed on the display 18 should be shapedin order to correspond to the actual location of the change in directionas shown on the video feed.

It will be understood that the navigation device 10 may also use acombination of the embodiments discussed above. For instance, thenavigation device may use orientation and positioning measurements toroughly determine the position of the navigation directions on thedisplay 18 and use pattern recognition techniques to determine theposition of the navigation directions on the display 18.

It will be understood that many alternatives and variations to the abovementioned embodiments can be thought of. For instance, another featureis that indication of road names, traffic signs (e.g. one way, no entry,exit numbers, place names etc.), speed limits, speed cameras, and pointsof interest stored in device memory 12, 13, 14, 15 can also besuperimposed over the video feed—the spatial location of this ‘virtualsignage’ in a video frame can correspond to the real world feature thatthe virtual signage relates to. Hence, a speed limit (e.g. the text ‘30mph’) could be superimposed so that it appears to overlie or be part ofthe road surface of the road with the 30 mph speed limit. An iconrepresenting a specific kind of traffic sign could be superimposed overthe video stream so that it appears in the place that a real world signwould usefully appear.

Other kinds of virtual signage in addition to direction arrows 5 mayalso be stored in memory devices 12, 13, 14, 15. For example, iconsrelating to road names, traffic signs, speed limits, speed cameras, busstops, museums, house numbers or points of interest may be stored inmemory devices 12, 13, 14, 15. All of these can also be superimposedover the video feed, with a spatial location in the displayed video thatcorresponds to the real world feature that the virtual signage relatesto. Hence, the software could take the 2D map data from the navigationsoftware that included the location data for these real world features,and apply a geometrical transformation that causes them to be correctlylocated when superimposed in the video feed.

According to a further alternative, the pattern recognition techniquesmay also be arranged to recognise objects on the road, such as forinstance an other vehicle or truck. When such an object is recognized,the displayed route 4 may be shown as a dotted line, such as shown inFIG. 11. This provides an image that more easy to interpret by a user.

Fourth Embodiment

According to fourth embodiment the feed from the camera 24 and thenavigations directions, such as position arrow 3, route 4, arrow 5,points of interest (POI), roads, buildings, map data, e.g. vector dataare not superimposed, but shown on the display 18 in a combined way.

This combination may be achieved by dividing the display in a first partand a second part, where the first part displays the camera feed and thesecond part displays the navigations directions. However, thecombination may also be performed in time, i.e. the navigation devicemay be arranged to successively show the camera feed and the navigationdirection in turns. This may be accomplished by showing the camera feedfor a first period (e.g. 2 seconds) and next, showing the navigationdirections for a second period (e.g. 2 seconds). However, the navigationdevice may also provide the user with the option to switch between thecamera feed and the navigation directions at his desire.

Of course, more than one camera may be used. The user may be providedwith the option to switch from a first camera feed to a second camerafeed. The user may also choose to display more than one camera feed onthe display 18 at the same time.

According to a further alternative, the user may zoom in or out. Whenzooming out, more and more of the environment of the navigation device10 will become displayed on the display 18. It will be understood thatthe user may choose for instance a helicopter view, as shown in FIG. 2,including the position of the navigation device 10. Such a view providesan image of the navigation device 10 (or vehicle) seen from behind. Ofcourse, such a view can not be provided by the camera, being fixed onthe navigation device 10 or vehicle. Therefore, the navigation device 10may provide an image as shown in FIG. 12, where only part of the imageis the camera view, surrounded by map data and navigation directions.

While specific embodiments of the invention have been described above,it will be appreciated that the invention may be practiced otherwisethan as described. For example, the invention may take the form of acomputer program containing one or more sequences of machine-readableinstructions describing a method as disclosed above, or a data storagemedium (e.g. semiconductor memory, magnetic or optical disk) having sucha computer program stored therein. It will be understood by a skilledperson that any of the software components may also be formed as ahardware component.

The descriptions above are intended to be illustrative, not limiting.Thus, it will be apparent to one skilled in the art that modificationsmay be made to the invention as described without departing from thescope of the claims set out below.

1. A device, comprising: a processing unit; and a memory arranged tostore information having associated location data, the processing unitbeing arranged to receive a video feed from a camera and data from apositioning device indicative of the location of at least one of thedevice and camera, the processing unit being further arranged to displaya combination of a camera image from the video feed from the camera andvirtual signage on a display, the virtual signage comprising the storedinformation or an icon representative thereof, and the virtual signagebeing superimposed over the camera image with a spatial location in thedisplayed camera image that corresponds to the location data of thestored information, wherein the device is arranged to receive user-inputcalibration corrections to adjust the location of the virtual signagewith respect to the displayed camera image via a device menu.
 2. Thedevice of claim 1, wherein the device is a navigation devicepositionable in a vehicle.
 3. The device of claim 1, wherein the deviceis a handheld device and the camera is formed integrally with thedevice.
 4. The device of claim 3, wherein the processing unit is furtherarranged to receive readings from at least one orientation sensor formedintegrally with the device, and to use the readings to compute anorientation of the device.
 5. The device of claim 4, wherein thecomputed orientation is used by the processing unit to superimpose thevirtual signage over the camera image.
 6. The device of claim 1, whereinthe virtual signage relates to one or more of: roads; road names;traffic signs; speed limits; speed cameras; buildings and points ofinterest.
 7. The device of claim 1, wherein the processing unit isfurther arranged to receive readings from at least one orientationsensor, and to use the readings to compute an orientation of at leastone of the device and camera.
 8. The device of claim 7, wherein thecomputed orientation is used by the processing unit to superimpose thevirtual signage over the camera image.
 9. The device of claim 1, whereinthe device is arranged to receive calibration corrections, to storethese calibration corrections, and to apply the calibration correctionswhen superimposing the virtual signage over the camera image.
 10. Anon-transitory computer readable medium comprising instructions which,when executed on a processing unit of a device, cause the processingunit to: receive a video feed from a camera and data from a positioningdevice indicative of the location of at least one of the device andcamera; access information stored on a memory of the device, saidinformation having associated location data; display a combination of acamera image from the video feed from the camera and virtual signage ona display, the virtual signage comprising the stored information or anicon representative thereof, and the virtual signage being superimposedover the camera image with a spatial location in the displayed cameraimage that corresponds to the location data of the stored information;and receive user-input calibration corrections to adjust the location ofthe virtual signage with respect to the displayed camera image via adevice menu.